A Dual-Iterative Method for Concept-Word Acquisition from Large-Scale Chinese Corpora

This paper proposes a dual-iterative method, a hierarchical inner and outer iteration method (HIO), to acquire concept words from a large-scale, un-segmented Chinese corpus. It has two levels of iteration: the EM-CLS algorithm and the Viterbi-C/S algorithm constitute the inner iteration for generating concept words, and the concept word validation constitutes the outer iteration together with the concept word generation. Through multiple iterations, it integrates the concept word generation and validation into a uniform acquisition process. In the process of acquisition, the HIO method can cope with the problem of over-segmentation, over-combination and data sparseness. The experimental result shows that the HIO method is valid for concept word acquisition that can simultaneously increase the precision and recall rate of concept word acquisition.